Semantic Segmentation to Identify and Treat Plants in a Field and Verify the Plant Treatments

ABSTRACT

A farming machine including a number of treatment mechanisms treats plants according to a treatment plan as the farming machine moves through the field. The control system of the farming machine executes a plant identification model configured to identify plants in the field for treatment. The control system generates a treatment map identifying which treatment mechanisms to actuate to treat the plants in the field. To generate a treatment map, the farming machine captures an image of plants, processes the image to identify plants, and generates a treatment map. The plant identification model can be a convolutional neural network having an input layer, an identification layer, and an output layer. The input layer has the dimensionality of the image, the identification layer has a greatly reduced dimensionality, and the output layer has the dimensionality of the treatment mechanisms.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/126,842, filed Sep. 10, 2018 which claims the benefit of U.S.Provisional Application No. 62/676,259, filed May 24, 2018 which isincorporated herein by reference in its entirety for all purposes.

BACKGROUND Field of Disclosure

This disclosure relates to using a plant identification model toidentify and treat plants in a field and, more specifically, using asemantic segmentation model to identify and treat plants in a field.

Description of the Related Art

Historically, farming machines that spray crops with treatment fluidhave relied on highly non-specific spray techniques such as broadcastspraying. Non-specific spray techniques are inefficient, wasteful, andcan be harmful to field health. More recently, non-specific spraying hasbeen supplemented with target-specific spray techniques that utilizedetection devices to determine plants in the field. However, even theseimproved spray techniques can be wasteful because their algorithms oftensacrifice accuracy, specificity, and resolution to identify plantsquickly. Accordingly, a farming machine used for targeted spraying thatuses an algorithm to rapidly identify plants without sacrificingaccuracy, specificity, and resolution would be beneficial.

SUMMARY

A farming machine can include any number of treatment mechanisms totreat plants according to a treatment plan as the farming machine movesthrough the field. A treatment plan identifies which plants in a fieldto treat and how to treat them. Each treatment mechanism can controlledby a control system that actuates the treatment mechanisms at theappropriate time to treat plants as the farming machine moves throughthe field. The farming machine can also include multiple detection andverification systems to capture images of plants in the field tofacilitate treating plants according to the treatment plan.

The control system of the farming machine can execute a plantidentification model configured to identify plants in the field fortreatment according to the treatment plan. The control system cangenerate a treatment map using the plant identification model. Thetreatment map is a data structure that includes information regardingwhich treatment mechanisms to actuate such that the treatment mechanismstreat identified plants according to the treatment plan.

To generate a treatment map, the farming machine captures an image ofplants in the field with a detection system. The control system accessesthe image and inputs the image to the plant identification model. Theplant identification model processes the image to identify plants in theimage according to the treatment plan. The plant identification modelgenerates a treatment map that maps treatment areas of the treatmentmechanisms to areas in the image including identified plants. Thecontrol system converts the treatment map into control signals for thetreatment mechanisms and actuates the treatment mechanisms at theappropriate time such that the identified plants are treated accordingto the treatment plan.

The plant identification model can be a convolutional neural networkincluding any number of nodes organized in any number of layers. In oneexample, the plant identification model executes a pixelwise semanticsegmentation (“semantic segmentation”) model on the neural network toidentify plants in accessed images and generate treatment maps. In thiscase, the plant identification model includes an input layer, anidentification layer, and an output layer. Each layer can be connectedby any number of additional hidden layers.

The plant identification model inputs the accessed image at the inputlayer. Generally the input layer has a dimensionality similar to thedimensionality of the accessed image. The plant identification modelreduces the dimensionality of the image using any number of functions.The functions can have any number of weights and parameters and act toidentify latent information in the accessed image that represent plants.The functions reduce the accessed image to the identification layer. Theidentification layer is a data structure that represents the types andcharacteristics of objects identified by the functions of the plantsemantic segmentation model.

The plant identification model outputs a treatment map at the outputlayer. Generally, the output layer has a dimensionality similar to thedimensionality of the plant treatment mechanisms of the farming machine.Again, the plant identification model can use any number of functions,weights, and hidden layers to increase the dimensionality from theidentification layer to the output layer and generate a treatment map.

In various configurations, the identification layer can be used toidentify any number of objects and conditions other than plants, suchas: the result of a plant treatment, different types of plants, thecondition of a plant, soil, weather, etc. Therefore, the plantidentification model can be used to execute treatment plans using any ofthe identified objects. For example, in one configuration, the plantidentification model can be configured to identify the result ofprevious plant treatments. As such, generated treatment map can be usedto verify previously executed plant treatments.

BRIEF DESCRIPTION OF DRAWINGS

The disclosed embodiments have advantages and features which will bemore readily apparent from the detailed description, the appendedclaims, and the accompanying figures (or drawings). A brief introductionof the figures is below.

Figure (FIGS. 1A and 1B are side and front views of a system forapplying a treatment fluid to plants in a field, according to oneexample embodiment.

FIG. 2 is an illustration of an image accessed by the control system,according to one example embodiment.

FIG. 3A-3E are visual representations of several example treatmentarrays for farming machines with various plant treatment mechanisms,according to some example embodiments.

FIG. 4 illustrates an access image with a treatment array mapped to itspixels, according to one example embodiment.

FIG. 5 illustrates an example treatment map, according to one exampleembodiment.

FIGS. 6A and 6B illustrate a mapped image and a treatment map generatedfrom the mapped image using a traditional bounding box method toidentify and treat plants with a farming machine, according to oneexample embodiment.

FIGS. 7A and 7B illustrate a mapped image and a treatment map generatedfrom the mapped using a semantic segmentation method to identify andtreat plants with a farming machine, according to one exampleembodiment.

FIG. 8 is a representation of a plant identification model based onaccessed images and previously identified plants, according to oneexample embodiment.

FIG. 9 is a flow diagram illustrating a method for the identificationand treatment of weeds in a cotton field, according to one exampleembodiment.

FIG. 10A is an illustration of an accessed image used in the method ofFIG. 9, according to one example embodiment.

FIG. 10B is an illustration of a treatment map generated from the in themethod of FIG. 9, according to one example embodiment.

FIG. 11 is a flow diagram illustrating a method for verifying a planttreatment in a field, according to one example embodiment.

FIG. 12A is an illustration of the accessed post-image in the method ofFIG. 11, according to one example embodiment.

FIG. 12B is an illustration of an executed treatment map generated fromthe in the method of FIG. 11, according to one example embodiment.

FIGS. 13A-13F illustrate several example dissimilarities that a controlsystem can determine by comparing a treatment map to an executedtreatment map, according to some example embodiments.

FIG. 14 is a block diagram illustrating components of an example machinefor reading and executing instructions from a machine-readable medium,according to one example embodiment.

DETAILED DESCRIPTION I. Introduction

Farming machines that treat plants in a field have continued to improveover time. For example, a crop sprayer can include a large number ofindependently actuated spray nozzles to spray treatment fluid onspecific plants in a field. The farming can further include any numberof detection mechanisms that can detect plants in the field and anytreatments made to plants in that field. Recently, farming machines haveincluded control systems executing algorithms to automatically detectand treat plants using the detection mechanisms. Traditionally, thealgorithms are wasteful in that they treat areas in the field that donot include identified plants because, often, the algorithms sacrificeaccuracy for processing speed.

Described herein is a farming machine that employs a semanticsegmentation model that automatically determines, in real-time, plantsin a field and treats the identified plants using a treatment mechanism.The semantic segmentation model encodes an image of the field to aconvolutional neural network trained to reduce the encoded image andidentify plants in the field. Rather than decoding the identified plantsback to an image, the semantic segmentation model decodes the identifiedplants to a treatment map which the farming machine uses to treat theplants in the field. The dimensionality of the treatment map is,generally, much less than the dimensionality of the image and,therefore, the processing time is reduced. The semantic segmentationmodel has higher accuracy, specificity, and provides better resolutionfor the treatment mechanisms than other traditional plant identificationmodels.

II. Plant Treatment System

FIG. 1A is a side view illustration of a system for applying a treatmentfluid to plants in a field and FIG. 1B is a front view illustration ofthe same system. The farming machine 100 for plant treatment includes adetection mechanism 110, a treatment mechanism 120, and a control system130. The farming machine 100 can additionally include a mountingmechanism 140, a verification mechanism 150, a power source, digitalmemory, communication apparatus, or any other suitable component.

The farming machine 100 functions to apply a treatment to one ormultiple plants 102 within a geographic area 104. Often, treatmentsfunction to regulate plant growth. The treatment is directly applied toa single plant 102 (e.g., hygroscopic material), but can alternativelybe directly applied to multiple plants, indirectly applied to one ormore plants, applied to the environment associated with the plant (e.g.,soil, atmosphere, or other suitable portion of the plant environmentadjacent to or connected by an environmental factor, such as wind), orotherwise applied to the plants. Treatments that can be applied includenecrosing the plant, necrosing a portion of the plant (e.g., pruning),regulating plant growth, or any other suitable plant treatment.Necrosing the plant can include dislodging the plant from the supportingsubstrate 106, incinerating a portion of the plant, applying a treatmentconcentration of working fluid (e.g., fertilizer, hormone, water, etc.)to the plant, or treating the plant in any other suitable manner.Regulating plant 102 growth can include promoting plant growth,promoting growth of a plant portion, hindering (e.g., retarding) plantor plant portion growth, or otherwise controlling plant growth. Examplesof regulating plant 102 growth includes applying growth hormone to theplant, applying fertilizer to the plant or substrate 106, applying adisease treatment or insect treatment to the plant, electricallystimulating the plant, watering the plant, pruning the plant, orotherwise treating the plant. Plant growth can additionally be regulatedby pruning, necrosing, or otherwise treating the plants adjacent theplant.

The plants 102 can be crops, but can alternatively be weeds or any othersuitable plant. The crop may be cotton, but can alternatively belettuce, soy beans, rice, carrots, tomatoes, corn, broccoli, cabbage,potatoes, wheat or any other suitable commercial crop. The plant fieldin which the system is used is an outdoor plant field, but canalternatively be plants within a greenhouse, a laboratory, a grow house,a set of containers, a machine, or any other suitable environment. Theplants are grown in one or more plant rows (e.g., plant beds), whereinthe plant rows are parallel, but can alternatively be grown in a set ofplant pots, wherein the plant pots can be ordered into rows or matricesor be randomly distributed, or be grown in any other suitableconfiguration. The crop rows are generally spaced between 2 inches and45 inches apart (e.g. as determined from the longitudinal row axis), butcan alternatively be spaced any suitable distance apart, or havevariable spacing between multiple rows.

The plants 102 within each plant field, plant row, or plant fieldsubdivision generally includes the same type of crop (e.g. same genus,same species, etc.), but can alternatively include multiple crops (e.g.,a first and a second crop), both of which are to be treated. Each plant102 can include a stem, arranged superior (e.g., above) the substrate106, which supports the branches, leaves, and fruits of the plant. Eachplant can additionally include a root system joined to the stem, locatedinferior the substrate plane (e.g., below ground), that supports theplant position and absorbs nutrients and water from the substrate 106.The plant can be a vascular plant, non-vascular plant, ligneous plant,herbaceous plant, or be any suitable type of plant. The plant can have asingle stem, multiple stems, or any number of stems. The plant can havea tap root system or a fibrous root system. The substrate 106 is soil,but can alternatively be a sponge or any other suitable substrate.

The treatment mechanism 120 of the farming machine 100 functions toapply a treatment to the identified plant 102. The treatment mechanism120 includes a treatment area 122 to which the treatment mechanism 120applies the treatment. The effect of the treatment can include plantnecrosis, plant growth stimulation, plant portion necrosis or removal,plant portion growth stimulation, or any other suitable treatmenteffect. The treatment can include plant 102 dislodgement from thesubstrate 106, severing the plant (e.g., cutting), plant incineration,electrical stimulation of the plant, fertilizer or growth hormoneapplication to the plant, watering the plant, light or other radiationapplication to the plant, injecting one or more working fluids into thesubstrate 106 adjacent the plant (e.g., within a threshold distance fromthe plant), or otherwise treating the plant. The treatment mechanism 120is operable between a standby mode, wherein the treatment mechanism 120does not apply a treatment, and a treatment mode, wherein the treatmentmechanism 120 is controlled by the control system 130 to apply thetreatment. However, the treatment mechanism 120 can be operable in anyother suitable number of operation modes.

The farming machine 100 can include a single treatment mechanism 120, orcan include multiple treatment mechanisms. The multiple treatmentmechanisms can be the same type of treatment mechanism, or be differenttypes of treatment mechanisms. The treatment mechanism 120 can be fixed(e.g., statically coupled) to the mounting mechanism 140 or relative tothe detection mechanism 110, or actuate relative to the mountingmechanism 140 or detection mechanism 110. For example, the treatmentmechanism 120 can rotate or translate relative to the detectionmechanism 110 and/or mounting mechanism 140. In one variation, thefarming machine 100 includes an assembly of treatment mechanisms,wherein a treatment mechanism 120 (or subcomponent of the treatmentmechanism 120) of the assembly is selected to apply the treatment to theidentified plant 120 or portion of a plant in response to identificationof the plant and the plant position relative to the assembly. In asecond variation, the farming machine 100 includes a single treatmentmechanism, wherein the treatment mechanism is actuated or the farmingmachine 100 moved to align the treatment mechanism 120 active area 122with the targeted plant 102. In a third variation, the farming machine100 includes an array of treatment mechanisms 120, wherein the treatmentmechanisms 120 are actuated or the farming machine 100 is moved to alignthe treatment mechanism 120 active areas 122 with the targeted plant 102or plant segment.

In one example configuration, the farming machine 100 can additionallyinclude a mounting mechanism 140 that functions to provide a mountingpoint for the system components. In one example, as shown in FIG. 1A,the mounting mechanism 140 statically retains and mechanically supportsthe positions of the detection mechanism 110, the treatment mechanism120, and the verification mechanism 150 relative to a longitudinal axisof the mounting mechanism 140. The mounting mechanism 140 is a chassisor frame, but can alternatively be any other suitable mountingmechanism. In some configurations, there may be no mounting mechanism140, or the mounting mechanism can be incorporated into any othercomponent of the farming machine 100.

In one example farming machine 100, the system may also include a firstset of coaxial wheels, each wheel of the set arranged along an opposingside of the mounting mechanism 140, and can additionally include asecond set of coaxial wheels, wherein the rotational axis of the secondset of wheels is parallel the rotational axis of the first set ofwheels. However, the system can include any suitable number of wheels inany suitable configuration. The farming machine 100 may also include acoupling mechanism 142, such as a hitch, that functions to removably orstatically couple to a drive mechanism, such as a tractor, more to therear of the drive mechanism (such that the farming machine 100 isdragged behind the drive mechanism), but alternatively the front of thedrive mechanism or to the side of the drive mechanism. Alternatively,the farming machine 100 can include the drive mechanism (e.g., a motorand drive train coupled to the first and/or second set of wheels). Inother example systems, the system may have any other means of traversingthrough the field.

In some example systems, the detection mechanism 110 can be mounted tothe mounting mechanism 140, such that the detection mechanism 110traverses over a geographic location before the treatment mechanism 120traverses over the geographic location. In one variation of the farmingmachine 100, the detection mechanism 110 is statically mounted to themounting mechanism 140 proximal the treatment mechanism 120. In variantsincluding a verification mechanism 150, the verification mechanism 150is arranged distal the detection mechanism 110, with the treatmentmechanism 120 arranged there between, such that the verificationmechanism 150 traverses over the geographic location after treatmentmechanism 120 traversal. However, the mounting mechanism 140 can retainthe relative positions of the system components in any other suitableconfiguration. In other systems, the detection mechanism 110 can beincorporated into any other component of the farming machine 100.

In some configurations, the farming machine 100 can additionally includea verification mechanism 150 that functions to record a measurement ofthe ambient environment of the farming machine 100, which is used toverify or determine the extent of plant treatment. The verificationmechanism 150 records a measurement of the geographic area previouslymeasured by the detection mechanism 110. The verification mechanism 150records a measurement of the geographic region encompassing the planttreated by the treatment mechanism 120. The verification mechanismmeasurement can additionally be used to empirically determine (e.g.,calibrate) treatment mechanism operation parameters to obtain thedesired treatment effect. The verification mechanism 150 can besubstantially similar (e.g., be the same type of mechanism as) thedetection mechanism 110, or be different from the detection mechanism110. The verification mechanism 150 can be a multispectral camera, astereo camera, a CCD camera, a single lens camera, hyperspectral imagingsystem, LIDAR system (light detection and ranging system), dyanmometer,IR camera, thermal camera, humidity sensor, light sensor, temperaturesensor, or any other suitable sensor. In other configurations of thefarming machine 100, the verification mechanism 150 can be included inother components of the system.

In some configurations, the farming machine 100 can additionally includea power source, which functions to power the system components,including the detection mechanism 100, control system 130, and treatmentmechanism 120. The power source can be mounted to the mounting mechanism140, can be removably coupled to the mounting mechanism 140, or can beseparate from the system (e.g., located on the drive mechanism). Thepower source can be a rechargeable power source (e.g., a set ofrechargeable batteries), an energy harvesting power source (e.g., asolar system), a fuel consuming power source (e.g., a set of fuel cellsor an internal combustion system), or any other suitable power source.In other configurations, the power source can be incorporated into anyother component of the farming machine 100.

In some configurations, the farming machine 100 can additionally includea communication apparatus, which functions to communicate (e.g., sendand/or receive) data between the control system 130 and a set of remotedevices. The communication apparatus can be a Wi-Fi communicationsystem, a cellular communication system, a short-range communicationsystem (e.g., Bluetooth, NFC, etc.), or any other suitable communicationsystem.

III. Generating a Treatment Map

A farming machine 100 can operate in a field according to a treatmentplan. A treatment plan determines which identified objects in the fieldare treated by the treatment mechanisms 120 of the farming machine 100.For example, a treatment plan can indicate the farming machine to treatweeds with herbicides, plants with growth promoters, soil withfungicides, etc. Whatever the treatment plan, as the farming machine 100travels through the field, the detection mechanism 110 identifies theobjects and treatment mechanisms 120 treats the objects.

In some cases, a farming machine 100 generates a treatment map tofacilitate treating identified objects in the field. Broadly, atreatment map is a data structure including information which representswhich treatment mechanisms 120 of a farming machine are actuated toappropriately treat plants in a field according to a treatment plan asthe farming machine is travelling through the field. However, rapidlyand accurately generating a treatment map while a farming machine 100travels through the field is a challenging problem. For context,successfully treating a plant requires a farming machine 100 to acquirean image with a detection mechanism 110, access the image, process theimage to identify plants according to a treatment plan, generate atreatment map based on the identified plants, and treat the plants basedon the treatment map by actuating appropriate plant treatment mechanismsas they pass above the identified plants. Executing all of these stepsin the amount of time it takes for the farming machine 100 to travelpast an identified plant in a field is non-trivial. In particular,processing images to quickly and precisely identify plants and generatea treatment map according to a treatment plan is especially challenging.

Generating a treatment map relies on a system controller 130 accessingan image of the field, identifying plants in the accessed image, andmapping the identified plants to treatment mechanisms 120 of the farmingmachine for treatment. This section describes generating a treatment mapby accessing an image and mapping treatment mechanisms to the accessedimage. The subsequent two sections describe identifying objects in anaccessed image and how that information can be used to generate atreatment map.

III.A Accessed Images

A farming machine 100 obtains images of a field using a detectionmechanism 110 as the farming machine 100 travels through the field and,generally, the obtained images include the treatment areas 122 of thetreatment mechanisms 120. Each obtained image also includes informationthat represents various features and objects in the field. For example,an image can include information representing a plant, a planttreatment, soil, field conditions, etc. The information can includecolor, shapes, sizes, metadata of the image, detection mechanismscharacteristics, pixel information, etc. The control system 130 of thefarming machine 100 can access and process the image to determine thefeatures and objects in the field using the information included in theimage. Based on the determined features and objects, the farming machine100 can execute various actions as described herein.

FIG. 2 is an illustration of an image accessed by the control system(i.e., accessed image 210). The accessed image 210 is obtained by adetection mechanism 110 coupled to a farming machine 100 as the farmingmachine 100 travels through a field. The accessed image 210 includesinformation representing a single plant of a first type 220, threeplants a second type 230 that are different sizes, and soil 240 in thefield.

The accessed image 210 has a pixel dimensionality. The pixeldimensionality is the number of pixels in the accessed image (e.g., 512pixels×1024 pixels, etc.). In some cases, the control system 130 canadjust the image to change the pixel dimensionality or information inthe image. For example, the control system 130 can change the pixeldimensionality by cropping, up-sampling, down-sampling, etc. as desired.Each pixel in the accessed image 210 represents an area of the field.The amount of area of the field that each pixel represents can depend onthe characteristics of the detection mechanism 110 (e.g., distance fromthe camera to the field, focal length, etc.). For example, each pixel inan accessed image can represent a 1 mm×1 mm area of the field. Stillfurther, groups of pixels in the accessed image can correspond to atreatment area 122 of a treatment mechanism 120. For example, if eachpixel represents a 1 mm×1 mm area and a treatment area 122 for atreatment mechanism is 1 cm×5 cm, the treatment area 122 can representby 10 pixels×50 pixels. That is, a group of pixels has a density (e.g.,pixels/cm) that relates an area (e.g., pixels²) in an accessed image 210to the treatment area 122 (e.g., cm²) in the field. In some cases, thecontrol system 130 stores information the amount of area of the field isassociated with each pixel. In other examples, the control system 130may store information regarding the configuration of the detectionmechanism 110 and farming machine 100 such that the control system candetermine the amount of area associated with each pixel.

III.B Treatment Arrays

Each farming machine 100 includes an array of treatment mechanisms 120.The system controller 130 can represent the treatment mechanisms 120 ofthe farming machine 100 as a treatment array. The treatment array is adata structure that includes a number of treatment elements with eachtreatment element of the treatment array corresponding to a treatmentmechanism 120 of the array of treatment mechanisms. Each treatmentelement, and the treatment array, are a representation of the number,orientation, size, configuration, etc. (hereinafter “dimensionality,” inaggregate) of the treatment mechanisms 120 of the farming machine 100.As such, a treatment array can be used by the control system 130 whenidentifying and treating objects in the field because each element ofthe treatment array has a real-world treatment mechanism 120counterpart. As a contextual example, a farming machine 100 includestreatment mechanisms 120 that are a single row of 30 adjacent spraynozzles. Therefore, the treatment array is a data structure whosedimensionality is 1×30 where each treatment element of the treatmentarray represents a single nozzle. Each treatment element in thetreatment array can include information regarding each treatmentelements associated nozzle (e.g., flow rate, size, etc.).

FIG. 3A-3E are visual representations of several example treatmentarrays 310 for farming machines 100 with various plant treatmentmechanisms 120. Each treatment element 320 of a treatment array 310corresponds to a treatment mechanism 120 of the farming machine 100.Notably, the dimensionality of some treatment arrays 310 are morecomplex and can correspond to more complex treatment mechanisms 120 of afarming machine 100. Some treatment arrays 310 include treatmentelements 320 with variable heights and widths, include spacing betweenelements, include treatment elements 320 arranged in uniqueorientations, etc. Various treatment arrays 310 are shown to illustratethat the dimensionality of a treatment array 310 can be variable and isnot intended to be limiting. Further, treatment arrays 310, includingboth their structure and the information included in each treatmentelement 320, can change over time (e.g., swapping nozzles, changing flowrates, etc.) or remain constant. FIG. 3A illustrates the treatmentmechanisms 120 associated with the treatment elements 320 of a treatmentarray 310 while the treatment elements 320 of the treatment arrays 310in FIGS. 3B-3E do not.

III.C Treatment Areas

Each plant treatment mechanism 120 treats a treatment area 122 in thefield when the farming machine 100 actuates the treatment mechanism.Thus, because each treatment element 320 in a treatment array 310 isassociated with a treatment mechanism 120, each treatment element 320can also be associated with a treatment area 122. For example, a farmingmachine 100 includes a single row of thirty identical treatmentmechanisms 120 thatt treat a fixed amount of treatment area 122. Assuch, the farming machine's 100 associated treatment array 310 has adimensionality of 1×30. Each treatment mechanism 120 treats a treatmentarea 122 in the field that is 10 cm×10 cm when actuated. Thus, eachtreatment element 320 in the treatment array 310 represents a 10 cm×10cm treatment area 122 in the field such that the treatment array 310represents an aggregate treatment area 122 of 10 cm×300 cm.

In some configurations, the size and shape of a treatment area 122 and,correspondingly a treatment element 320, may be a function of farmingmachine characteristics. For example, the treatment area 122 can bebased on the speed of the farming machine 100, direction of the farmingmachine 100, orientation of the treatment mechanism 120, the size of thetreatment mechanism 120, flow conditions of a treatment fluid, amount oftime the treatment mechanism 120 is actuated, etc.

For example, a treatment mechanism 120 is a nozzle that sprays treatmentfluid. The nozzle is oriented such that the spray pattern of thetreatment fluid is a horizontal line orthogonal to the direction ofmovement off the farming machine 100. In this example, the nozzle spraypattern is approximately 2.5 cm wide, the treatment time off the nozzlesis 0.1 s, and the velocity of the farming machine is 2.5 m/s.Accordingly, the treatment area 122 for the farming machine traveling atthis velocity can be approximated with the following the equation:

A=v·w·t  (1)

where A is the treatment area 122, v is the velocity, w is the treatmentwidth, and t is the treatment time. In other configurations, thetreatment area 122 can be approximated by other equations.

In this example, the treatment area 122 for the spray nozzle isapproximately 6.25 mm². Assuming that the treatment time is the minimumcontrollable treatment time for the nozzle, 6.25 mm² is the minimumtreatment area 122 for the spray nozzle when the farming machine istravelling at 2.5 m/s. If the spray nozzle is a single nozzle of a 1×30array of spray nozzles, then the aggregate treatment area 122 is 18.7cm². In some configurations, each treatment element 320 of a treatmentarray 310 is associated with minimum treatment area 122 of a treatmentmechanism 120. For example, the preceding description of a farmingmachine 100 including treatment mechanisms 120 with fixed treatmentareas 122, those treatment areas 122 may be the minimum treatment areas122 of the farming machine 100 due to limitations of the treatmentmechanism 120 or limitations to actuation of the treatment mechanisms120.

In various embodiments, a treatment area 122 can be calculated by thecontrol system 130 of the farming machine 100. For example, the controlsystem 130 can access information regarding any of the aforementionedfactors used to determine the treatment area 122 and determine thetreatment area 122 based on that information.

III.D Mapping Treatment Arrays to Accessed Images

A treatment array 310 can be mapped to an accessed image 210 becausegroups of pixels representing real-world areas in an accessed image 210can be configured to correspond to a treatment area 122 of a treatmentmechanism 120. As such, if a real-world area corresponds to a treatmentarea 122 of a treatment mechanism 120, the treatment element 320 of atreatment array 310 corresponding to that treatment mechanism 120 canalso correspond to that real-world area.

For example, a treatment array 310 is 1×8 treatment elements 320 witheach treatment element corresponding to a treatment mechanism 120 with atreatment area 122 of 5 cm×10 cm. Each pixel in an accessed imagecorresponds to 1 mm×1 mm. Therefore, each treatment element 320 of thetreatment array 310 corresponds to 50 pixels×100 pixels of the accessedimage 210 and the treatment array 310 corresponds to 50 pixels by 1000pixels. Accordingly, the control system 130 can map the treatment array310 to an accessed image 210. The control system 130 can align themapping of the treatment array to the accessed image in a variety ofmanners. In one example, the center of the accessed image 210 cancorrespond to the center of the array of treatment mechanisms 120 of thefarming machine 100. In another examples, the control system 130 mayalign the mapping by cross-referencing calibration between a detectionmechanism 110 and a verification mechanism 150 with a keypoint detectorthat matches images taken from different viewpoints, estimates cameraheight from ground and estimates a pixel density (in pixels per inch).In yet another example, the control system may cross-reference treatmentmechanism commands to detected treatments in the accessed image with apixelwise segmentation network configured for detecting treatments.

FIG. 4 illustrates an accessed image 210 with a treatment array 310mapped to its pixels (mapped image 410). FIG. 4 shows the accessed image210 of FIG. 2 with an array of treatment areas 122 mapped to the mappedimage 410. The treatment areas 122 are illustrated as a dashed lineswithin the mapped image 410. Each treatment area 122 in the imagecorresponds to a treatment element 320 of a treatment array 310. In thisexample, the treatment areas 122 correspond to a treatment array 310similar to the treatment array 310 of FIG. 3B. Because each treatmentarea 122 is mapped to a set of pixels in the mapped image 410, thevarious treatment areas 122 in the mapped image 410 include informationrepresenting any of a plant of the first type 320, a plant of the secondtype 330, and soil 340 of the field.

The treatment areas 122 in FIG. 4 can also correspond to the treatmentarray 310 of FIG. 3A mapped to the accessed image 210 of FIG. 3. In thiscase, a farming machine 100 is moving through the field and each row oftreatment areas 122 in the mapped image 410 illustrates how thetreatment array 310 progresses through the field as the farming machine100 travels. That is, at a first time t₁ the treatment areas 122 of thetreatment mechanisms 120 are located at the first row. At a second timet₂ the farming machine 100 has moved through the field and the treatmentareas 122 are now located at the second row, etc.

In various embodiments, mapping a treatment array 310 to an accessedimage 210 can occur at various times. In one example, a control system130 may map a treatment array 310 to an accessed image 210 and thenidentify plants in the image. In another example, a control system 130may identify plants in an image and the map a treatment array 310 to theidentified plants. Whatever the configuration, identified plants can beassociated with treatment elements 320 of the treatment array 310 suchthat the farming machine 100 can appropriately treat plants withtreatment mechanisms 120.

II.E Treatment Maps

The control system 130 generates a treatment map to control planttreatment by treatment mechanisms 120 of a farming machine 100. Broadly,a treatment map indicates which treatment mechanisms 120 of a farmingmachine 100 are actuated to appropriately treat plants in a fieldaccording to a treatment plan. More specifically, a treatment map is atreatment array 310 mapped to an accessed image 210 and includesinformation regarding which treatment elements 320 of the treatmentarray 310 include identified objects. Therefore, a treatment map issimilarly structured to a treatment array and has the treatmentdimensionality.

A treatment map includes map elements that each correspond to treatmentareas 122 of a treatment mechanism 120. Further, as shown herein, eachmap element in a treatment map can correspond to a treatment area 122 inan accessed image 210. In a treatment map, each map element may alsoinclude information as to what objects are identified in thecorresponding treatment areas 122 of an accessed image 210. For example,a control system 130 accesses an image 210 and processes the informationin that image to determine pixels indicating plants. The control system130 associates the identified plant pixels of the accessed image 210 tomap elements of a treatment map including the identified pixels.

FIG. 5 illustrates an example treatment map 510. Here, the treatment map510 includes a similar structure to the treatment array 310 of FIG. 3E.In this example, the control system 130 processes an accessed image 210and identifies plants in the accessed image 210. The control system 130maps the treatment elements 320 of the treatment array 310 to theaccessed image 210. The control system 130 generates a treatment map byassociating identified plants to treatment elements 320 of a treatmentarray 310, thereby generating map elements 520 of a treatment map 510.In this example, the identified plants are associated with the selectedmap elements 530 (illustrated as shaded map elements) at the center ofthe treatment map 510. The control system 130 can actuate the treatmentmechanisms 120 associated with the selected map elements 530 (orunselected map elements) to treat treatment areas 122 based on atreatment plan.

IV. Plant and Object Identification Methods

As described above, processing images to quickly and precisely identifyplants according to a treatment plan is a challenging problem.Oftentimes, a farming machine 100 sacrifices identification accuracy forprocessing speed and vice versa. Traditionally, there have been severalmethods implemented to identify plants within an accessed image 210.Some example methods include: bounding box methods, color identificationmethods, normalized difference vegetation index based methods, etc. Onedrawback of traditional plant identification methods is theoverestimation of necessary treatment areas 122 in a treatment plan todecrease processing time.

FIGS. 6A and 6B illustrate a mapped image 410 and a treatment map 510generated from the mapped image 410 using a traditional bounding boxmethod to identify and treat plants with a farming machine 100. Eachillustrated treatment area 122 corresponds to a treatment element 320 ofa treatment array 310.

The bounding box method may overestimate the treatment areas 122 fortreating plants according to a treatment plan. For example, a farmingmachine 100 executes a treatment plan to selectively identify and treata first type of plant 260 in a field with a growth promoter from atreatment mechanism 120. The method identifies pixels within a mappedimage 410 that are likely to include a plant of the first type 320 andplaces them within a bounding box 310. For example, referring to themapped image 410 of FIG. 6A, the bounding box 310 is generallyrectangular or square in shape and includes pixels indicating a plant ofthe first type 320. Also, in this example, the edges of a bounding box610 are mapped to the edges of treatment areas 122, but may be mapped toany other area in the accessed image 210.

A control system 130 generates a treatment map 510 by mapping thebounding box 320 and its constituent identified object(s) to mapelements 520 of a treatment map 510. That is, the control system 130associates identified objects to treatment elements 320 of the treatmentarray 310 of the mapped image 410 that include those objects to generatea treatment map 510.

FIG. 6B illustrates a treatment map 510 generated from the bounding box610 and mapped image 410 of FIG. 6A. The treatment map 510 includes bothmap elements and selected map elements 330 (shaded). In this example,selected map elements 530 correspond to treatment areas 122 of theaccessed image 210 bounded by the bounding box 320. The selectedtreatment elements 530 will be treated by the treatment mechanisms 120of the farming machine 100. Notably, because the bounding box 610 wasrectangular, the selected map elements 330 also form a rectangle.Because the bounding box 610 conformed to a particular shape that wassubsequently used to generate a treatment map 510, the selectedtreatment elements 530 include inaccurate map elements 532 (indicated bya ‘*’). Inaccurate map elements 532 correspond to treatment areas 122having only soil 340 or soil and the second type of plant 270.

In this example, the farming machine 100 actuates the treatmentmechanisms 120 associated with the selected map elements 530 of thetreatment map 510 and treats the treatment areas 122 corresponding tothe selected map elements 530 with a growth promoter. The identifiedplant of the first type 260 is treated with a growth promoter asintended. However, a plant of the second type 270 is unintentionallytreated with the growth promoter and, additionally, areas of the fieldnot including the first type of plant 260 (e.g., only soil 280) aretreated with the growth promoter. As such, in this example, the boundingbox method can be wasteful and detrimental to field health.

V. Semantic Segmentation

Pixelwise semantic segmentation is a general identification method that,when applied to the problem of identifying plants, may be faster andmore accurate than other plant identification methods. Pixelwisesemantic segmentation models are a subset of deep learning methods thatoperate on fully convolutional encoder-decoder network. FIGS. 7A and 7Billustrate a mapped image 410 and a treatment map 510 generated from themapped image 410 using a semantic segmentation method to identify andtreat plants with a farming machine 100. Each illustrated treatment area122 corresponds to a treatment element 320 of a treatment array 310.

In this example, the farming machine 100 is configured to identify thefirst type of plant 260 and treat the first type of plant 260 with agrowth promoter. The control system 130 identifies groups of pixels in amapped image 410 as the first type of plant 260 using semanticsegmentation. The control system 130 similarly identifies groups ofpixels representing the second type of plant 270 in the mapped image.For example, referring to the mapped image 410 of FIG. 7A, the mappedimage 410 illustrates groups of pixels likely to represent plants of afirst type 260 with a first type of fill and groups of pixels likely torepresent plants of a second type 270 with a second type of fill.Contrary to the bounding box method, the identified groups of pixels cantake any shape and are not limited to a bounding box.

FIG. 7B illustrates a treatment map 510 generated from pixels identifiedas the first type of plant 260 in the mapped image 410 using semanticsegmentation. In this example, selected map elements 530 are treatmentareas 122 corresponding to all treatment areas 122 including pixels inthe accessed mapped 410 identified as the first type of plant 260.Because the identified pixels can take any size or shape, the treatmentmap 510 no longer includes selected map elements 530 that are inaccuratetreatment map elements 532. Therefore, the farming machine 100 onlyapplies growth treatment to treatment areas 122 including the firstplant type 260.

Semantic segmentation can be implemented by a control system 130 using aplant identification model. A farming machine 100 can execute the plantidentification model to identify features (e.g., plants) in an accessedimage 210 and quickly generate an accurate treatment map 510. FIG. 8 isa representation of a plant identification model 800 based on accessedimages 210 and previously identified plants. The previously identifiedplants may have been identified by another plant identification model ora human identifier.

In the illustrated embodiment, referred to throughout the remainder ofthe specification, the plant identification model 800 is a convolutionalneural network model with layers of nodes, in which values at nodes of acurrent layer are a transformation of values at nodes of a previouslayer. A transformation in the model is determined through a set ofweights and parameters connecting the current layer and the previouslayer. For example, as shown in FIG. 8, the example model 800 includesfive layers of nodes: layers 810, 820, 830, 840, and 850. Thetransformation from layer 810 to layer 820 is given by W₁, thetransformation from layer 820 to layer 830 is given by W₂, thetransformation from layer 830 to layer 840 is given by W₃, and thetransformation from layer 840 to layer 850 is given by W₄. In someexamples, the transformation can also be determined through a set ofweights and parameters used to transform between previous layers in themodel. For example, the transformation W₁ ^(T) from layer 840 to layer850 can be based on parameters used to accomplish the transformation W₁from layer 810 to 820.

The input to the model 800 is an accessed image 210 (or a mapped image410) encoded onto the convolutional layer 810 and the output of themodel is a treatment map 510 decoded from the output layer 850. Themodel 800 identifies latent information in the accessed imagerepresenting plants in the identification layer 830. The model 800reduces of the dimensionality of the convolutional layer 810 to that ofthe identification layer 830 to identify the plants, and then increasesthe dimensionality of the identification layer 830 to generate atreatment map 510.

The accessed image 210 is encoded to a convolutional layer 810. In oneexample, accessed image is directly encoded to the convolutional layer810 because the dimensionality of the convolutional layer 810 is thesame as the pixel dimensionality of the accessed image 210. In otherexamples, the accessed image 210 can be adjusted such that thedimensionality of the accessed image 210 is the same as thedimensionality of the convolutional layer 810.

Accessed images 210 in the convolutional layer 810 can be related toplant identification information in the identification layer 830.Relevance information between these elements can be retrieved byapplying a set of transformations between the corresponding layers.Continuing with the example from FIG. 8, the convolutional layer 810 ofthe model 800 represents an encoded accessed image 210, andidentification layer 830 of the model 800 represents plantidentification information. Plants in a given accessed image 210 may beidentified by applying the transformations W₁ and W₂ to the pixel valuesof the accessed image 210 in the space of convolutional layer 810. Theweights and parameters for the transformations may indicaterelationships between information contained in the accessed image andthe identification of a plant. For example, the weights and parameterscan be a quantization of shapes, colors, etc. included in informationrepresenting plants included in an accessed image 210. The weights andparameters may be learned historical user interaction data includingplant identification submitted by users.

In one example, the weights and parameters can be collected and trainedusing data collected from previously accessed images 210 and a labellingprocess. The labelling process reduces the amount of time required bysegmentation model 800 to identify pixels representing plants while alsoincreasing the accuracy of identified pixels. The labelling process caninclude: identifying a bounding box pixels including pixels likely torepresent a plant, identifying green pixels within the bounding boxes,identifying a contour around the identified green pixels, using thecontours to create a label set for the accessed image, and sending thelabelled image to users for verification. Users verify labels byidentifying which pixels are green near the contours in an accessedimage rather than an entire image. In effect, this “pre-identification”by model 800 reduces the number of pixels need to be verified by a humanand reduces the costs of training the semantic segmentation model 800.

Plants in the accessed image 210 are identified in the identificationlayer 830. The identification layer 830 is a data structure representingidentified plants based on the latent information about the plantsrepresented in the accessed image 210. The identification layer has adimensionality less than the pixel dimensionality and treatmentdimensionality.

Identified plants in an accessed image 210 can be used to generate atreatment map 510. To generate a treatment map, the model 800 starts atthe identification layer 830 and applies the transformations W₂ and W₃to the value of the given identified plants in the identification layer830. The transformations result in a set of nodes in the output layer850. The weights and parameters for the transformations may indicaterelationships between an identified plants and a treatment map 510. Insome cases, the plant treatment map 510 is directly output from thenodes of the output layer 850, while in other cases the control systemdecodes the nodes of the output layer 850 into a treatment map 510. Thatis, model 800 can include a conversion layer (not illustrated) thatconverts the output layer 850 to a treatment map 510.

Take, for example, a model 800 the produces an output image with anarbitrary set of pixels marked to indicate the presence of a plant atoutput layer 850. A conversion layer can convert the output image into aset of boxes that generate a treatment map 510. Most generally, theconversion layer receives a two dimensional array (e.g., an outputimage) as input and outputs a list of quadruples representing a box.Each box of the list is defined such that pixels indicating anidentified plant in the output image approximately match the area of thebox. Each box can be associated with a treatment area 122 such thatfarming machine 100 treats plants identified in the accessed image 210that are represented in the output image. In some examples, theconversion layer is governed by a policy that sets a value of a box to a0 or 1 according to the treatment plan. For example, if the treatmentplan is to treat an identified plant the policy may set a value for abox to 1. Further, the conversion layer can include a treatmentresolution. The treatment resolution represents the smallest size of abox that can correspond to a treatment area 122 of a treatment mechanism120. Take, for example, a conversion layer including a treatmentresolution of 8 pixels. In this case, if an output image includes a setof pixels indicating a plant that is 6 pixels, the conversion layer cancreate a box that is 8 pixels rather than 6 pixels. The conversion layercreates a larger box because a box representing 6 pixels is below thetreatment resolution.

Here, the output of the model 800 is dissimilar to that of a traditionalencoder/decoder scheme. Generally, the output layer 850 of a traditionalconvolutional neural network has the same, or nearly the same,dimensionality as the convolutional layer 810. That is, given theexample of an accessed image 210 as an input, the convolutional layer810 and the output layer 850 both have the pixel dimensionality. Thisallows a convolutional neural network to identify objects in an accessedimage 210 and map them back to the original input accessed image 210.Traditionally, decoding objects identified in the identification layer830 to an image having the same dimensionality as the convolutionallayer 810 is computationally expensive and requires additionalprocessing time. In this example, the dimensionality of the treatmentmap 510 is the treatment dimensionality of a farming machine 100.Generally, the treatment dimensionality (e.g., 1×30 treatment mechanisms120) is much less than the pixel dimensionality (e.g., 2880×3840 pixels)and, therefore, decoding the identified plants in the identificationlayer 830 to a treatment map 510 requires fewer transformations and lessprocessing time.

Additionally, the model 800 can include layers known as intermediatelayers. Intermediate layers are those that do not correspond to anaccessed image 210, plant identification, or a treatment map 510. Forexample, as shown in FIG. 4, layers 820 are intermediate encoder layersbetween the convolutional layer 810 and the identification layer 830.Layer 840 is an intermediate decoder layer between the identificationlayer 830 and the output layer 850. Hidden layers are latentrepresentations of different aspects of a plant that are not observed inthe data, but may govern the relationships between the elements of animage when identifying a plant. For example, a node in the hidden layermay have strong connections (e.g., large weight values) to input valuesand identification values that share the commonality of “green leaves.”As another example, another node in the hidden layer may have strongconnections to input values and identification values that share thecommonality of “sprayed ground.” Specifically, in the example model ofFIG. 4, nodes of the hidden layers 820 and 840 can link inherentinformation in the accessed image that share common characteristics tohelp determine if that information represents a plant in the accessedimage.

Additionally, each intermediate layer is a combination of functions suchas, for example, residual blocks, convolutional layers, poolingoperations, skip connections, concatenations, etc. Any number ofintermediate encoder layers 820 can function to reduce the convolutionallayer to the identification layer and any number of intermediate decoderlayers 840 can function to increase the identification layer 830 to theoutput layer 850. Alternatively stated, the encoder intermediate layersreduce the pixel dimensionality to the identification dimensionality,and the decoder intermediate layers increase the identificationdimensionality to the treatment dimensionality.

In one particular example, an intermediate layer can include a dice losslayer for semantic segmentation. The function of a dice loss layer canbe defined as:

$\begin{matrix}{{{{softDiceCoeff}\left( {p,g} \right)} = \frac{2\Sigma_{i}p_{i}*g_{i}}{{\Sigma_{i}p_{i}} + {\Sigma_{i}g_{i}}}}{and}} & (1) \\{{{Dice}\mspace{14mu} {Loss}} = {1 - {softMeanDiceCoeff}}} & (2)\end{matrix}$

where p, is the probability of outputs of a softmax layer and g is theground truth. In equation 2, the “mean” in softMeanDiceCoeff indicatesthat the dice coefficient may calculated for numerous classes in themodel and then averaged.

Furthermore, in various embodiments, the functions of the model 800 canreduce the accessed image 210 and identify any number of objects in afield. The identified objects are represented in the identificationlayer 830 as a data structure having the identification dimensionality.In various example, the model can be configured to identify a locationof a plant, a condition of a plant, a type of a plant, a category of aplant (e.g., a weed or a crop), or any other characteristics of a plantin the field. In various other embodiments, the identification layer canidentify latent information representing other objects in the accessedimage. For example, the identification layer 830 can identify a resultof a plant treatment, soil, an obstruction, or any other object in thefield.

VI. Treatment Maps to Control Treatments of a Farming Machine

A treatment map 510 can be used by a farming machine 100 to treat plantsin a field. As previously described, a treatment map 510 includes anynumber of map elements 520. Each map element 520 in a treatment map 510is associated with a treatment area 122 in an accessed image 210. Eachmap element 520 can also include information regarding identifiedobjects in the treatment areas 122 associated with that map element 520.As such, a treatment map 510 can be used to control treatment mechanisms120 of a farming machine 100 to appropriately treat plants according toa treatment plan.

For example, a control system 130 can input a treatment map 510 andoutput control signals. The control signals are sent to treatmentmechanisms 120 of the farming machine 100 to actuate the treatmentmechanisms 120. Generally, the control signals are sent at anappropriate time such that the treatment mechanism 120 is actuated whilethe treatment mechanism 120 is above the identified object.

As an example, a treatment map is a 1×30 data structure where each mapelement 510 is associated with a spray nozzle. Here, a map element 520of the treatment map 510 indicates that its associated treatment area122 in an accessed image 210 contains pixels indicative of a weed. Assuch, the control system 130 generates control signals for the treatmentmechanism(s) 120 associated with the map element 520 such that thetreatment mechanism 120 treats the weed as the farming machine 100travels over the weed in the field.

VII. Plant Identification to Control Treatments of a Farming Machine

A farming machine 100 can execute a plant identification model (e.g.,model 800) to identify plants in a field. The plant identification model800 inputs an accessed image 210 of the field and generates a treatmentmap 510. The treatment map 510 includes selected map elements 530associated with treatment areas 122 in the field to be treated withtreatment mechanisms 120. The selected map elements 530 are thetreatment areas 122 in the accessed image 210 that include an identifiedplants and, more generally, an area for treatment. The farming machine100 treats the identified plants based on the treatment map 510 byactuating treatment mechanisms 120 to treat the identified plants.

FIGS. 9 and 10 illustrate a specific example of a farming machine 100identifying and treating plants using plant treatment mechanisms 120 asthe farming machine 100 travels through the field. In this example, thefarming machine 100 is a crop sprayer operating in a field planted withcotton (e.g., first plant type 260). The treatment plan for the cropsprayer is to identify weeds (e.g. second plant type 270) in the fieldand treat them by spraying them with an herbicide. The farming machine100 is configured with a single row of eight spray nozzles that serve astreatment mechanisms 120. That is, the spray nozzles spray an herbicidewhen actuated by the farming machine 100. The farming machine 100includes a detection mechanism 110 that captures images of plants in thefield as the farming machine 100 travels down the cotton crop rows.Further, the farming machine 110 includes a control system 130 thatidentifies plants in the field and controls the spray nozzles.

FIG. 9 is a flow diagram illustrating a method 900 for theidentification and treatment of weeds in a cotton field. The method 900can include additional or fewer steps and the illustrated steps may beaccomplished in any order. In some cases, steps may be repeated anynumber of times before progressing to a subsequent step.

To begin, the farming machine 100 images an area of the field using thedetection mechanism 110. The image includes information representingcotton plants, weed plants, and soil in the field. In this example, thedetection mechanism 110 is mounted to the front of the farming machine100 such that the area of the field is imaged before the front end ofthe farming machine 100 passes over the area. The detection mechanism110 transmits the image to the control system 130 of the farming machine100.

The control system accesses 910 the image from the detection mechanismand accesses the information included therein. FIG. 10A is anillustration of the accessed image 210. The accessed image 210 includesa cotton plant 1010 near the center of the image and several weeds 1020spaced about the cotton plant. For convenience, the treatment areas 122of a treatment array for the farming machine 100 are overlaid on theaccessed image 250. In this case the treatment array 420 is an array ofspray nozzles 1030 and the treatment areas 122 are spray areas 1040.Spray nozzles 1030 labeled 1 to 8 are pictured above the accessed image210 and correspond to the treatment areas 122 in the columns below them.

In this example, the spray areas 1040 are shown for eight differenttreatments of the spray nozzles 630 from time t₁ through t₈. Thus, eachrow of illustrated spray areas 1040 corresponds to one treatment of thespray areas 1040 in that row by the farming machine 100 at a particulartime. Each illustrated treatment area 122 is the minimum treatment area122 for the corresponding spray nozzle 1030. Over time, as the farmingmachine 100 moves through the field, the spray nozzles 1030 and theircorresponding spray areas 1040 move down the accessed image 210.

Returning to FIG. 9, the control system applies 920 a model 800 toidentify pixels in the accessed image 210 representing weeds 1020. Thecontrol system 130 encodes the accessed image 210 onto a convolutionallayer 810 of the model 800. Each pixel of the accessed image 210corresponds to a single node of the convolutional layer 810 such thatthe pixel dimensionality of the accessed image 210 and dimensionality ofthe convolutional layer 810 are the same.

The model 800 applies a set of transformations to the convolutionallayer 810 to identify weeds 1020 in the accessed image 210. Eachtransformation identifies latent information in the pixels thatrepresents weeds 1020 and reduces the dimensionality of convolutionallayer 810. The transformations reduce the convolutional layer 810 to theidentification layer 830. The identification layer 830 is configured toidentify weeds, cotton, and soil and, therefore, the identificationdimensionality is 1×3.

The control system 130 generates 930 a treatment map 510 from theidentification layer 830. The model 800 applies a set of transformationsto the identification layer 830 to generate the treatment map 510. Eachtransformation increases the dimensionality of the identification layer830 and decodes the identified plants to a new dimensionality. The setof transformations increases the dimensionality of the identificationlayer to that of the output layer 850.

The output layer 450 has the same dimensionality as the treatmentdimensionality (e.g., 1×8) such that the output layer 450 can representa treatment map. That is, each node of the output layer corresponds to amap element 520 of a treatment map 510 and, thereby, a spray area 1040of a spray nozzle 1030 of the farming machine 100. Further, each nodealso includes information regarding objects identified in that sprayarea 1040 (e.g., weed, cotton, or soil). As such, the control system 130can decode the nodes of the output layer into a treatment map. In thisexample, the control system 130 generates a treatment map 510 indicatingthat for map element 520 of a treatment map 510 including an identifiedweed, the farming machine 100 actuates the corresponding spray nozzle1030 to spray herbicide in the appropriate spray area 1040.

FIG. 10B illustrates a treatment map 510 generated by the model 800. Thetreatment map 510 includes map elements 520 corresponding to similarlypositioned spray areas 1040 of FIG. 6A. The treatment map 510 alsoincludes selected map elements 530 which correspond to spray areas 1040that included information representing a weed 1020 in the accessed image210. Selected map elements 530 will be treated by the farming machine100 as it travels through the field.

Returning to FIG. 9, the farming machine 100 actuates 940 spray nozzles630 to spray the spray areas 1040 and treat the weeds 1020 as thefarming machine 100 travels through the field. To do so, the controlsystem 130 generates control signals for the spray nozzles 1030 andactuates the spray nozzles 1030 corresponding to selected map elements530 areas in the treatment map 510 at the appropriate time.

For example, turning to FIGS. 6A and 6B, at time t₁ the treatment map510 indicates a selected map element 530 in the 7^(th) column becausethe corresponding spray area 1040 for the 7^(th) spray nozzle 1030 attime t₁ included pixels representing a weed 1020. As such, the farmingmachine actuates the 7^(th) spray nozzle 1030 at time t₁ to treat theweed 1020 in the corresponding spray area 1040 as the farming machine100 travels through the field.

The process continues as the farming machine 100 travels through thefield. As the farming machine 100 moves, the model 800 generates theappropriate treatment maps 510 to spray herbicide on weeds 1020 and notcotton 1010. For example, continuing from time t₁, the farming machine100 actuates the 7^(th) treatment mechanism 120 at time t₂, the 3^(rd)treatment mechanism 120 at time t₃, the 1^(st) treatment mechanism 120at time t₆, the 1^(st) and 2^(nd) treatment mechanism at time t₇, andthe first treatment mechanism at time t₈.

VIII. Semantic Segmentation for Plant Treatment Verification

A farming machine 100 can also be used to verify a plant treatment. Forexample, rather than accessing an image obtained by the detectionmechanism 110, the control system can access an image obtained by theverification mechanism 150 (a “post-image”). The post-image includesboth information regarding objects in the field and informationregarding any plant treatments executed by the treatment mechanisms 120of the farming machine 100 (a “treated area”). For example, thepost-image may include information regarding a treated area that hasbeen sprayed by a spray nozzle.

A post-image can be used to verify treatments by a treatment plan. To doso, the control system 130 accesses a post-image, determines thelocation of plant treatments in the post-image, generates an executedtreatment map, and compares the executed treatment map to the treatmentmap.

FIGS. 11 and 12 illustrate a specific example of a farming machine 100verifying plant treatments by treatment mechanisms 120 of a farmingmachine 100 as the farming machine 100 travels through a field. In thisexample, the farming machine 100 is a crop sprayer operating in a fieldplanted with cotton 1010. The treatment plan for the crop sprayer is toidentify cotton in the field and spray them with a fertilizer. Thefarming machine 100 is similarly configured to the farming machine 100in the example of FIGS. 9-10.

FIG. 11 is a flow diagram illustrating a method 1100 for verifying aplant treatment in a field. The method 1100 can include additional orfewer steps and the illustrated steps may be accomplished in any order.In some cases, steps may be repeated any number of times beforeprogressing to a subsequent step.

To begin, the farming machine 100 executes 1110 a plant treatment plan.That is, the farming machine 100 images an area of the field using thedetection mechanism 110, identifies cotton plants in the field,generates a treatment map 510 based on the identified plants, andactuates the spray nozzles to spray fertilizer on the cotton plants asthe farming machine 100 travels through the field. The generatedtreatment map is stored on a storage system of the farming machine 100.

Next, the framing machine 100 captures a post-image of the area usingthe verification mechanism 150. The post-image includes informationrepresenting the treated areas, cotton plants, weed plants, and soil inthe field. In this example, the verification mechanism 150 is mounted tothe rear of the farming machine 100 such that the area of the field isimaged after the spray nozzles pass over the area and treat the cottonplants. The verification mechanism 150 transmits the post-image to thecontrol system 130 of the farming machine 100.

The control system 130 accesses 1120 the post-image from theverification mechanism 150 and accesses the information includedtherein. FIG. 12A is an illustration of the accessed post-image 212. Theaccessed post-image 212 includes a cotton plant 1010 near the center ofthe image and several weeds 1020 spaced about the cotton plant 1010. Theaccessed image 210 also includes information representing a treated areawhere certain spray areas 1040 have been sprayed by the spray nozzles.In this case, the information indicating a plant treatment isillustrated as dampened soil 1210.

Returning to FIG. 11, the control system applies 1130 a model 800 toidentify pixels in the accessed post-image 212 representing a treated.In this case, the model 800 is configured to identify treated areasrather than plants. That is, the control system 130 encodes the accessedpost-image 212 onto a convolutional layer 810 of the model 800. Themodel 800 applies a set of transformations to the convolutional layer810 to identify a treated area in the accessed post-image 212. Thetransformations reduce the convolutional layer 810 to the identificationlayer 830. The identification layer 830 is configured to identifydampened soil 1210 as treated areas.

The control system 130 applies 1140 the model to generate an executedtreatment map 510 from the identification layer 830. The executedtreatment map indicates which spray areas 1040 include information thatindicates that they were previously sprayed by a spray nozzle.

FIG. 12B illustrates an executed treatment map 512 generated by themodel 800. The executed treatment map 512 includes map elements 520corresponding to similarly positioned spray areas 1040 of FIG. 12A. Theexecuted treatment map 512 also includes selected map elements 530 whichcorrespond to spray areas 1040 that included information representing aa treated area (e.g., dampened soil 1210) in the accessed post-image212.

The control system 130 verifies a plant treatment for a particular areaby determining a dissimilarity between a treatment map 510 and anexecuted treatment map 512 for the particular area. If an executedtreatment map 512 is the same as a treatment map 510, then the farmingmachine 100 is appropriately treating plants as it passes by them in thefield. If the executed treatment map 512 is dissimilar to the treatmentmap 510, then the farming machine 100 may not be appropriately treatingplants as it passes them in the field.

Once the control system determines a dissimilarity, the control systemcan determine a source for the dissimilarity. Once the source of thedissimilarity is identified, the control system 130 can notify anoperator of the farming machine or take actions to correct the source ofthe dissimilarities. Correcting the source of the dissimilarities caninclude re-calibrating spatial awareness of control system 130 usingmeasurements of detection system 110 or verification system 150 asground control points, or notifying an operator of the farming machine100.

FIGS. 13A-13F illustrate several example dissimilarities that a controlsystem 130 can determine by comparing a treatment map 510 to an executedtreatment map 512. Each executed treatment map 512 in FIGS. 13A-13F arecompared to the treatment map 510 of FIG. 7B to determinedissimilarities.

FIG. 13A is a first example executed treatment map that is dissimilarfrom a treatment map 510. In this example, the map elements in thefourth column are all shifted down by one map element relative to thetreatment map 510 of FIG. 7B. This can indicate that there is a delay inactuating a treatment mechanism corresponding to the map elements of thefourth column as the farming machine travels through the field.

FIG. 13B is a second example executed treatment map 512 that isdissimilar from a treatment map 510. In this example, the model isconfigured to determine varying degrees of plant treatment and,accordingly, an executed treatment map can indicate varying degrees ofplant treatment. For example, the model can identify spray areas thathave been heavily sprayed or lightly sprayed. Here, the executedtreatment map 512 includes map elements in the fourth column thatindicate that the treatment areas 122 corresponding map elements in thefourth column are only lightly sprayed instead of heavily sprayed as inthe treatment map of FIG. 7B. Accordingly, this dissimilarity canindicate that the treatment mechanism associated with the fourth columnmay have a reduced flow rate.

FIG. 13C is a third example executed treatment map 512 that isdissimilar from a treatment map 510. In this example, the map elementsin the sixth column are all shifted right by one map element relative tothe treatment map 510 of FIG. 7B. This can indicate that there is amisalignment of the treatment mechanism corresponding to the mapelements of the sixth column (e.g., a spray nozzle spraying off axis).

FIG. 13D is a fourth example executed treatment map 512 that isdissimilar from a treatment map 510. In this example, the map elementsin the fifth column are missing relative to the treatment map 510 ofFIG. 7B. This can indicate that the treatment mechanism corresponding tothe map elements of the fifth column is clogged.

FIG. 13E is a fifth example executed treatment map 512 that isdissimilar from a treatment map 510. In this example, the map someelements in the fifth column are missing relative to the treatment map510 of FIG. 7B. This can indicate that the treatment mechanismcorresponding to the map elements of is intermittently actuating.

FIG. 13F is a sixth example executed treatment map 512 that isdissimilar from a treatment map 510. In this example, the map someelements are rotated ninety degrees relative to the treatment map 510 ofFIG. 7B. This can indicate that the farming machine is travelling in thewrong direction. In various embodiments, the control system candetermine any number of other angles of rotation.

The proceeding examples in FIGS. 13A-F are given as means of an example.The control system 130 can determine any other dissimilarity bycomparing a treatment map to an executed treatment map. Also, note thatthe illustrated discrepancies are limited by the size of the treatmentareas 122 relative to the size of the treatment maps. Generally,reducing the size of the treatment areas 122 relative to a treatment mapincreases the capability of the control system to determinediscrepancies.

IX. Control System

FIG. 14 is a block diagram illustrating components of an example machinefor reading and executing instructions from a machine-readable medium.Specifically, FIG. 14 shows a diagrammatic representation of controlsystem 130 in the example form of a computer system 1400. The computersystem 1400 can be used to execute instructions 1424 (e.g., program codeor software) for causing the machine to perform any one or more of themethodologies (or processes) described herein. In alternativeembodiments, the machine operates as a standalone device or a connected(e.g., networked) device that connects to other machines. In a networkeddeployment, the machine may operate in the capacity of a server machineor a client machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a smartphone, aninternet of things (IoT) appliance, a network router, switch or bridge,or any machine capable of executing instructions 1424 (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute instructions 1424 to perform any one or more of themethodologies discussed herein.

The example computer system 1400 includes one or more processing units(generally processor 1402). The processor 1402 is, for example, acentral processing unit (CPU), a graphics processing unit (GPU), adigital signal processor (DSP), a controller, a state machine, one ormore application specific integrated circuits (ASICs), one or moreradio-frequency integrated circuits (RFICs), or any combination ofthese. The computer system 1400 also includes a main memory 1404. Thecomputer system may include a storage unit 1416. The processor 1402,memory 1404, and the storage unit 1416 communicate via a bus 1408.

In addition, the computer system 1406 can include a static memory 1406,a graphics display 1410 (e.g., to drive a plasma display panel (PDP), aliquid crystal display (LCD), or a projector). The computer system 1400may also include alphanumeric input device 1412 (e.g., a keyboard), acursor control device 1414 (e.g., a mouse, a trackball, a joystick, amotion sensor, or other pointing instrument), a signal generation device1418 (e.g., a speaker), and a network interface device 1420, which alsoare configured to communicate via the bus 1408.

The storage unit 1416 includes a machine-readable medium 1422 on whichis stored instructions 1424 (e.g., software) embodying any one or moreof the methodologies or functions described herein. For example, theinstructions 1424 may include the functionalities of modules of thesystem 130 described in FIG. 2. The instructions 1424 may also reside,completely or at least partially, within the main memory 1404 or withinthe processor 1402 (e.g., within a processor's cache memory) duringexecution thereof by the computer system 1400, the main memory 1404 andthe processor 1402 also constituting machine-readable media. Theinstructions 1424 may be transmitted or received over a network 1426 viathe network interface device 1420.

X. Additional Considerations

In the description above, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofthe illustrated system and its operations. It will be apparent, however,to one skilled in the art that the system can be operated without thesespecific details. In other instances, structures and devices are shownin block diagram form in order to avoid obscuring the system.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the system. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

Some portions of the detailed descriptions are presented in terms ofalgorithms or models and symbolic representations of operations on databits within a computer memory. An algorithm is here, and generally,conceived to be steps leading to a desired result. The steps are thoserequiring physical transformations or manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Some of the operations described herein are performed by a computerphysically mounted within a machine 100. This computer may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a computer readable storage medium, such as, but is notlimited to, any type of disk including floppy disks, optical disks,CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), randomaccess memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, orany type of non-transitory computer readable storage medium suitable forstoring electronic instructions.

The figures and the description above relate to various embodiments byway of illustration only. It should be noted that from the followingdiscussion, alternative embodiments of the structures and methodsdisclosed herein will be readily recognized as viable alternatives thatmay be employed without departing from the principles of what isclaimed.

One or more embodiments have been described above, examples of which areillustrated in the accompanying figures. It is noted that whereverpracticable similar or like reference numbers may be used in the figuresand may indicate similar or like functionality. The figures depictembodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. It should be understood thatthese terms are not intended as synonyms for each other. For example,some embodiments may be described using the term “connected” to indicatethat two or more elements are in direct physical or electrical contactwith each other. In another example, some embodiments may be describedusing the term “coupled” to indicate that two or more elements are indirect physical or electrical contact. The term “coupled,” however, mayalso mean that two or more elements are not in direct physical orelectrical contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B is true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the system. This descriptionshould be read to include one or at least one and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for asystem and a process for identifying and treating plants with a farmingmachine including a control system executing a semantic segmentationmodel. Thus, while particular embodiments and applications have beenillustrated and described, it is to be understood that the disclosedembodiments are not limited to the precise construction and componentsdisclosed herein. Various modifications, changes and variations, whichwill be apparent to those, skilled in the art, may be made in thearrangement, operation and details of the method and apparatus disclosedherein without departing from the spirit and scope defined in theappended claims.

What is claimed is:
 1. A method for identifying a plant in a field, themethod comprising: accessing an image of the field captured by animaging system, the accessed image having a pixel dimensionality andincluding a plurality of pixels with a subset of the pixels representingthe plant in the field; applying a semantic segmentation model to theimage to identify which pixels of the image represent the plant, thesemantic segmentation model configured to: identify the subset of pixelsin the image representing the plant, generate a treatment map comprisinga plurality of elements, each element of the plurality corresponding toa group of pixels of the plurality of pixels, one or more of theelements corresponding to the subset of pixels representing the plant,and each element indicating a treatment for a portion of the fieldrepresented by pixels corresponding to that element, and wherein adimensionality of the plurality of elements in the treatment map is lessor equal to the pixel dimensionality.
 2. The method of claim 1, furthercomprising: storing the treatment map in a treatment map databaseaccessible by a farming machine.
 3. The method of claim 1, furthercomprising: treating the plant in the field according to the treatmentmap.
 4. The method of claim 1, wherein the farming machine comprises aplurality of treatment mechanisms for treating plants, the plurality oftreatment mechanisms having a treatment mechanism dimensionality that isthe same as the treatment map dimensionality.
 5. The method of claim 1,further comprising: for each element in the plant treatment map,generating a plant treatment instruction corresponding to the treatmentindicated by the element comprising the subset of pixels identified asrepresenting the plant; storing the plant treatment instructions in aninstruction database.
 6. The method of claim 5, further comprising:transmitting the plant treatment instructions to the farming machine,the farming machine comprising a plurality of plant treatment mechanismsconfigured to actuate the plant treatment mechanisms using the planttreatment instruction to treat the identified plant based on thetreatment map.
 7. The method of claim 1, wherein the semanticsegmentation model further comprises: a plurality of neural networklayers comprising: an input layer comprising a plurality of input nodes,a number of the plurality of input nodes corresponding to the pixeldimensionality, an output layer comprising a plurality of output nodes,a number of the plurality of output nodes corresponding to the treatmentmap dimensionality, and one or more intermediate layers comprising aplurality of intermediate nodes, the nodes of the intermediate layerstransforming data in the input layer to data in the output layer.
 8. Themethod of claim 1, wherein applying the semantic segmentation modelfurther comprises: accessing a treatment array including a plurality oftreatment elements, wherein: the treatment array has the treatmentdimensionality, each of the plurality of treatment elements correspondsto an element of the treatment map, and each of the plurality oftreatment elements corresponds to a treatment area of a plant treatmentmechanism of a farming machine.
 9. The method of claim 8, furthercomprising: mapping the treatment array to the accessed image such thateach element of the treatment array corresponds to a group of pixels inthe accessed image representing the treatment area of the correspondingtreatment mechanism.
 10. The method of claim 9, further comprising:mapping the identified plants to elements of the treatment array togenerate a treatment map such that each element of the treatment mapincludes the identified plant when the identified plant is in itscorresponding treatment area.
 11. The method of claim 1, wherein: thetreatment dimensionality is a representation of any of the number, size,shape, arrangement, and configuration of the treatments applied by aplurality of plant treatment mechanisms.
 12. The method of claim 1,wherein the farming machine is any of a crop sprayer, a tractor, atiller, or a seeder.
 13. The method of claim 1, wherein the treatment isa spray treatment from an array of spray nozzles.
 14. The method ofclaim 13, wherein the spray treatment is water, nutrients, a herbicide,a fungicide, or another pesticide.
 15. A non-transitory computerreadable storage medium comprising instructions for identifying a plantin a field, the instructions, when executed by one or more processors,causing the one or more processors to: access an image of the fieldcaptured by an imaging system, the accessed image having a pixeldimensionality and including a plurality of pixels with a subset of thepixels representing the plant in the field; apply a semanticsegmentation model to the image to identify which pixels of the imagerepresent the plant, the semantic segmentation model configured to:identify the subset of pixels in the image representing the plant,generate a treatment map comprising a plurality of elements, eachelement of the plurality corresponding to a group of pixels of theplurality of pixels, one or more of the elements corresponding to thesubset of pixels representing the plant, and each element indicating atreatment for a portion of the field represented by pixels correspondingto that element, and wherein a dimensionality of the plurality ofelements in the treatment map is less or equal to the pixeldimensionality.
 16. The non-transitory computer readable storage mediumof claim 15, wherein the instructions, when executed, cause the one ormore processors to: store the treatment map in a treatment map databaseaccessible by a farming machine.
 17. The non-transitory computerreadable storage medium of claim 15, wherein the instructions, whenexecuted, cause the one or more processors to: treat the plant in thefield according to the treatment map, the treatment map executed by afarming machine comprising a plurality of treatment mechanismsconfigured to receive and execute the treatment map.
 18. A farmingmachine comprising: a non-transitory computer readable storage mediumcomprising instructions for identifying a plant in a field, theinstructions, when executed by one or more processors, causing the oneor more processors to: access an image of the field captured by animaging system, the accessed image having a pixel dimensionality andincluding a plurality of pixels with a subset of the pixels representingthe plant in the field; apply a semantic segmentation model to the imageto identify which pixels of the image represent the plant, the semanticsegmentation model configured to: identify the subset of pixels in theimage representing the plant, generate a treatment map comprising aplurality of elements, each element of the plurality corresponding to agroup of pixels of the plurality of pixels, one or more of the elementscorresponding to the subset of pixels representing the plant, and eachelement indicating a treatment for a portion of the field represented bypixels corresponding to that element, and wherein a dimensionality ofthe plurality of elements in the treatment map is less or equal to thepixel dimensionality.
 19. The farming machine of claim 18, wherein theinstructions, when executed, cause the one or more processors to: storethe treatment map in a treatment map database accessible by a farmingmachine.
 20. The farming machine of claim 18, wherein the instructions,when executed, cause the one or more processors to: treat the plant inthe field according to the treatment map, the treatment map executed bya farming machine comprising a plurality of treatment mechanismsconfigured to receive and execute the treatment map.